An Elasticity Study of Distributed Graph Processing

Graphs are a natural fit for modeling concepts used in solving diverse problems in science, commerce, engineering, and governance. Responding to the variety of graph data and algorithms, many parallel and distributed graph-processing systems exist. However, until now these platforms use a static model of deployment: they only run on a pre-defined set of machines. This raises many conceptual and pragmatic issues, including misfit with the highly dynamic nature of graph processing, and could lead to resource waste and high operational costs. In contrast, in this work we explore a dynamic, elastic model of deployment. To conduct an in-depth elasticity study of distributed graph processing, we build a prototype, JoyGraph, which is the first such system that implements complex, policy-based, and finegrained elasticity. Using the state-of-the-art LDBC Graphalytics benchmark and the SPEC Cloud Group’s elasticity metrics, we show the benefits of elasticity in graph processing: improved resource utilization, and aligned operation-workload dynamicity. Furthermore, we explore the cost of elasticity in graph processing. We identify a key drawback: although elasticity does not degrade application throughput, graph-processing workloads are sensitive to data movement while leasing or releasing resources. I. PROBLEM STATEMENT AND CONCEPTUAL CONTRIBUTION Modern graph-processing systems use sophisticated techniques to exploit the massive parallelism of scale-up machines [1] or the large-scale resources of distributed systems [2]. However, the use of the static deployment model has negative consequences. Conceptually, graph applications are a poor fit for static, fixed-size infrastructure, as they have often iterative, but highly irregular workloads [3]. Previous techniques do not focus on a dynamic model of deployment, where infrastructure can grow and shrink to match the needs of the graph-processing application. In contrast, in this work we explore the use of elasticity, that is, the ability to lease and release machines dynamically, as a key-feature of a new class of graph-processing systems. Our study is the first to explore the implications of elastic scaling policies, both growing and shrinking resources, on distributed graph processing. We give important performance insights, and quantify the cost and benefits of elasticity in graph processing. For public infrastructure, elasticity could reduce operational costs by reducing resource waste [4], [5], and improve the ability to meet Quality-of-Service guarantees (such as high performance and availability) by using appropriate autoscaling policies [6]. For private infrastructure, elasticity could reduce operational costs by increasing resource utilization [7], or throttle throughput to meet demands across applications [8]. Typically, when making a case for elasticity [9] in graph processing, only graph-related metrics are considered, such as active vertices, or messages exchanged per super-step. Many complementary studies present an analysis on how the number of active vertices affects the algorithm efficiency: directionoptimizing BFS [10] is a technique entirely motivated by the variable number of active edges per iteration; Mizan [11] analyzes runtime per iteration and addresses the large imbalance; GraphReduce [12] leverages the observed variability in the number of active vertices (frontier size) for two datasets and three algorithms; high workload imbalance appears even between multiple designs and implementations of the same algorithm running the same workload [3]; etc. In contrast, we also analyze the impact of workload imbalance on the consumption of system-level resources. II. ELASTIC GRAPH PROCESSING SYSTEM For our exploratory experiments aiming to understand the performance of graph processing under elasticity, we use our prototype, JoyGraph, which adapts the classic mechanisms of elasticity to graph processing. We also propose three new autoscaling policies for deciding when to grow and shrink the infrastructure, and new mechanisms for nested data repartitioning without loss. JoyGraph consists of three key components: The Master node, which orchestrates the execution of the graph processing workloads on the worker nodes; Worker nodes, which execute super-steps of the graph processing workloads; and Storage, a distributed file system which is used to load the input graph and to store the generated results. JoyGraph supports a Pregellike programming model. Policy-based elasticity for graph processing has not been explored until now. Recently, we have seen that generic autoscaling policies perform well with scientific workloads, almost on-par with workload-specific policies [6]. Encouraged by such results, we apply autoscaling policies to graph processing. Therefore, for JoyGraph, we evaluate two types of elastic autoscaling policies: system-level policies, and generic autoscaling policies. The former are simple autoscaling policies that take into account system level metrics, such as CPU-load, wallclock time, and network-load. The latter are general autoscaling policies, originally used successfully for autoscaling web applications. We consider three system-level elastic-scaling policies: CPU Load Policy (CPU): The average worker CPU-load is used to generate a new partition function to redistribute graph vertices. Wallclock Time Policy (WCP): Computes the wallclock time of each worker on each super-step. Workers that take more time 0 4 7 superstep 0 200 400 600 time (s) 0 4 8 12 16 20 m ac hi ne s supply demand (a) BFS + WCP. 0 5 9 superstep 0 50